Joint-Attention Learning in Prosody Transfer Speech Synthesis
Demo
High-quality text-to-speech (TTS) synthesis has remained a challenging research topic for years. Pushing the edge of the general naturalness of the synthesized utterance, several state-of-the-art models such as Tacotron and DeepVoice3 achieve excellent results in improving the quality of synthesized speech. To aim at more realistic speech synthesis, prosody-flexible TTS, also called expressive TTS has recently becomes a topic of significant research. For example, Google has proposed an ex- pressive TTS framework to successfully learn a reference utter- ance’s prosody and transfer it to a new utterance synthesized by the system. In this paper, we propose a prosody transfer text- to-speech synthesis model. Our work is implemented based on the end-to-end CNN block-based model of Baidu’s DeepVoice3 (DV3). Different from former models, in our work, we use a joint-attention learning process of the reference prosody and text. This comparatively simpler model can learn the reference input’s prosody along with the text input. A token table and weights are also learned with the reference input to factorize the possible styles in an unsupervised manner. The results show our model can successfully factorize the reference prosodies to represent characteristics of different speakers and styles, under unsupervised learning from the training data.
Deep-Voice3 Model¶
Style Encoder and Tokens¶
In this paper, our prosody transfer TTS system is built on an open source Baidu’s DV3 system. The model is shown as following. Based on the DV3, We added in a reference encoder into the framework. The reference encoder learns the extracted weights directly from the reference audio. And the weights matrix will then be directly multiplied with the randomly initialized token table to give the combined token as reference embedding. In our model there is no explicit attention module used to learn the similarity between the reference audio’s feature with the global tokens table. In contrast, predicted weights are ex- tracted directly from the reference utterance input and combined with global token table to give a reference embedding for further usage. The learned reference embedding is inserted to the text encoder in the Encoder PreNet and Convolution Blocks, so that a joint learned (key, value) based on the text and reference style is feed into the attention block in the decoder.
Proposed model¶
1. Token factorization¶
Assigned tokens gave different speaker voice as synthesis results
Uterrance: "I’ve felt the chance that I have a number of options."¶
To listen, files are at following:
Assigned tokens gave different synthesis results
Uterrance: "Just recovered a fumble on ensuing kickoff."¶
To listen, files are at following:
Token factorization on Blizzard 2013 dataset
Uterrance: "Just recovered a fumble on ensuing kickoff."¶
To listen, files are at following:
2. Prosody Transfer¶
Parallel utterances
The following shows three example of prosody transfer synthesis.
In each example, text of the utterance to synthesis is the same as the reference's. The first utterance shown in each example is the reference. The second one is the synthesis results using neutral prosody. The third one is the prosody transfer result.
Example 1¶
Utterance text content: My mother always took him to the town on a market day in a light gig.
Example 2¶
Utterance text content: So we never saw Dick any more.
Example 3¶
Utterance text content: You will be to visit me in prison with a basket of provisions, you will not refuse to visit me in prison?
Unparallel utterance <\center><\h3>
The following shows three example of unparallel prosody transfer synthesis.
In each example, text of the utterance to synthesis is different from the reference's. The first utterance shown in each example is the reference. The second and third ones are two prosody transfer synthesis results with different text contents.
The following shows three example of unparallel prosody transfer synthesis.
In each example, text of the utterance to synthesis is different from the reference's. The first utterance shown in each example is the reference. The second and third ones are two prosody transfer synthesis results with different text contents.
Example 1¶
Reference text: My mother always took him to the town on a market day in a light gig.
Prosody Transfer result 1's text: So we never saw Dick any more.
Prosody Transfer result 2's text: Just recovered a fumble on ensuing kickoff.
The prosody of the unparallel reference utterance will be transfered to the synthesis results having different text contents.
Example 2¶
Reference text: You will be to visit me in prison with a basket of provisions, you will not refuse to visit me in prison?
Prosody Transfer result 1's text: My mother always took him to the town on a market day in a light gig.
Prosody Transfer result 2's text: There was nothing disagreeable in Mister Rushworth's appearance.
The prosody of the unparallel reference utterance will be transfered to the synthesis results having different text contents.
Example 3¶
Reference text: There was nothing disagreeable in Mister Rushworth's appearance, and Sir Thomas was liking him already.
Prosody Transfer result 1's text: Just recovered a fumble on ensuing kickoff.
Prosody Transfer result 2's text: My mother always took him to the town on a market day in a light gig.
The prosody of the unparallel reference utterance will be transfered to the synthesis results having different text contents.